Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 99.28%. Comparing base (c186db0) to head (8aa380b).
Report is 2 commits behind head on main.

@@           Coverage Diff           @@
##             main     #168   +/-   ##
=======================================
  Coverage   99.28%   99.28%           
=======================================
  Files          22       22           
  Lines        1534     1534           
=======================================
  Hits         1523     1523           
  Misses         11       11           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Hi, thanks for this PR! I haven't done this because of #146 . I'll get back to this once I fix that.

While we're at it, do you think it makes sense to give the option to the LLM? We could make this a boolean parameter that the LLM can call, so that it can decide whether it wants chunks or full documents. The downside of it is that the LLM doesn't know the project before the func call, so its decisions might be bad and unreliable.

Yeah, I don't think it makes sense to expose that directly to an LLM -- whether it makes sense to return the chunk or the document is really going to be dependent on the size of the full document responses. It's not something I would expect the LLM to have enough context to make an intelligent decision on.
I suppose we could have a max_response_size argument provided by the LLM and automatically switch from document response to chunks if the total document response is over that size. I'm not sure how self-aware most LLMs are about their own context sizes though. Maybe the max_response_size should be what's exposed through the plugin opts though? 🤔

I suppose we could have a max_response_size argument provided by the LLM and automatically switch from document response to chunks if the total document response is over that size.

We might be able to utilise codecompanion's token count feature for this. The problem is how we can determine the max context window so that the token count actually makes sense (for example, a 100k-token conversation is close to being saturated for a 128k LLM, but is still very usable for a 1M LLM).

I also have this thought about using an auxiliary LLM as a response-rewriter that paraphrases the document content into concise, descriptive paragraphs as the response. Paraphrasing, compared to coding, is a simpler task and can hopefully be handled well enough by a small, cheap (or free) LLM. If it works, it could save the token count for the main LLM, allowing users to have longer conversations with the main coder LLM, and hopefully reduce the cost as well.

I'll checkout from this branch to refactor some of the code to prepare the codebase for #179 . All your commits will stay. I'll also make the changes that I mentioned here.